Local tone mapping

ABSTRACT

Systems and methods are disclosed for image signal processing. For example, methods may include receiving an image from an image sensor; applying a filter to the image to obtain a low-frequency component image and a high-frequency component image; determining a first enhanced image based on a weighted sum of the low-frequency component image and the high-frequency component image, where the high-frequency component image is weighted more than the low-frequency component image; determining a second enhanced image based on the first enhanced image and a tone mapping; and storing, displaying, or transmitting an output image based on the second enhanced image.

CROSS REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. patent application Ser. No.15/690,772, filed Aug. 30, 2018, the disclosure of which is herebyincorporated by reference in its entirety.

TECHNICAL FIELD

This disclosure relates to local tone mapping.

BACKGROUND

Image capture devices, such as cameras, may capture content as images orvideo. Light may be received and focused via a lens and may be convertedto an electronic image signal by an image sensor. The image signal maybe processed by an image signal processor (ISP) to form an image, whichmay be stored and/or encoded. In some implementations, multiple imagesor video frames from different image sensors may include spatiallyadjacent or overlapping content, which may be stitched together to forma larger image with a larger field of view. The image stitching processmay introduce distortions that depend on the objects appearing withinthe field of view of the camera and/or the relative positions andorientations of those objects.

SUMMARY

Disclosed herein are implementations of local tone mapping.

In a first aspect, the subject matter described in this specificationcan be embodied in systems that include an image sensor configured tocapture an image. The systems include a processing apparatus configuredto receive the image from the image sensor; apply a filter to the imageto obtain a low-frequency component image and a high-frequency componentimage; determine a first enhanced image based on a weighted sum of thelow-frequency component image and the high-frequency component image,where the high-frequency component image is weighted more than thelow-frequency component image; determine a second enhanced image basedon the first enhanced image and a tone mapping; and store, display, ortransmit an output image based on the second enhanced image.

In a second aspect, the subject matter described in this specificationcan be embodied in methods that include receiving an image from an imagesensor; applying a filter to the image to obtain a low-frequencycomponent image and a high-frequency component image; determining afirst enhanced image based on a weighted sum of the low-frequencycomponent image and the high-frequency component image, where thehigh-frequency component image is weighted more than the low-frequencycomponent image; determining a second enhanced image based on the firstenhanced image and a tone mapping; and storing, displaying, ortransmitting an output image based on the second enhanced image.

In a third aspect, the subject matter described in this specificationcan be embodied in systems that include an image sensor configured tocapture an image. The systems include a processing apparatus configuredto receive the image from the image sensor; apply a filter to the imageto obtain a low-frequency component image and a high-frequency componentimage; apply a non-linear mapping to the low-frequency component imageto obtain gains for respective image portions; apply the gains forrespective image portions to corresponding image portions of the imageto obtain an enhanced image; and store, display, or transmit an outputimage based on the enhanced image.

In a fourth aspect, the subject matter described in this specificationcan be embodied in systems that include an image sensor configured tocapture an image. The systems include a processing apparatus configuredto receive the image from the image sensor; apply a bilateral filter tothe image to obtain a low-frequency component image and a high-frequencycomponent image; determine a first enhanced image based on a weightedsum of the low-frequency component image and the high-frequencycomponent image, where the high-frequency component image is weightedmore than the low-frequency component image; determine a second enhancedimage based on the first enhanced image and a tone mapping; determine aperceptual domain image based on the second enhanced image and a gammacurve that models human perception of contrast; determine alow-frequency component perceptual domain image and a high-frequencycomponent perceptual domain image as components of the perceptual domainimage; determine a third enhanced image based on a weighted sum of thelow-frequency component perceptual domain image and the high-frequencycomponent perceptual domain image, where the high-frequency componentperceptual domain image is weighted more than the low-frequencycomponent perceptual domain image; and store, display, or transmit anoutput image based on the third enhanced image.

These and other aspects of the present disclosure are disclosed in thefollowing detailed description, the appended claims, and theaccompanying figures.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosure is best understood from the following detaileddescription when read in conjunction with the accompanying drawings. Itis emphasized that, according to common practice, the various featuresof the drawings are not to-scale. On the contrary, the dimensions of thevarious features are arbitrarily expanded or reduced for clarity.

FIG. 1 is a diagram of an example of an image capture system for contentcapture.

FIG. 2A is a block diagram of an example system configured for imagecapture and tone mapping.

FIG. 2B is a block diagram of an example system configured for imagecapture and tone mapping.

FIG. 3 is a flowchart of an example technique for local tone mapping ofa captured image.

FIG. 4 is a flowchart of an example technique for local tone mapping ofa captured image.

FIG. 5 is a flowchart of an example technique for local tone mapping ofa captured image.

FIG. 6 is a flowchart of an example technique for high-frequencyenhancement with underflow handling.

DETAILED DESCRIPTION

This document includes disclosure of systems, apparatus, and methods forlocal tone mapping to enable enhancement of the quality of imagesgenerated by image capture systems. Tone mapping is a process ofadjusting image luminance to improve contrast. Tone mapping may becomposed of two parts: (1) a non-linear response that mimics the eyenon-linear response to luminance, which is independent to the imagecontent and may be known as a gamma curve γ( ); and (2) an imagedependent contrast enhancement that may be known as a tone curve λ( )(e.g., a tone curve that when applied to a specific image implementsluminance histogram equalization). For example, a gamma curve maytransform the luminance value v according to γ(v)=v{circumflex over( )}g, with g<1, so that dark shades are lightened. When a tone curve isindependent of the pixel location in the image, this may be known asglobal tone mapping. A problem with global tone mapping is that someparts of the image see their contrast lowered (e.g., the contrast ofbrighter regions of an image may be lowered).

To address the problem of reduced contrast caused by a global tonemapping, a local tone mapping may be applied that enhances contrastlocally. For example, a principle of a local tone mapping may be todetermine components of an input image, including a low-frequencycomponent image (e.g., a base layer of the image or a smooth version ofthe image) and a high-frequency component image (e.g., details of theimage, which may be a compliment of a base layer), to enable differenttreatment of these components for enhancement of details and/or localcontrast. For example, a low-frequency component image (e.g., a baselayer image) may be determined from the image using an un-sharp maskfilter or a bilateral filter. In some implementations, details and/orcontrast are enhanced by multiplying the high-frequency component image(e.g., the compliment of a base layer) by a constant α>1 and adding itback to the low-frequency component to obtain an enhanced image. In someimplementations, local contrast is preserved by applying a global tonemapping gains selected based on the low-frequency component image bothto pixels of the low-frequency component image (e.g., a base layer) andto pixels of the high-frequency component image (e.g., details).

Details and/or contrast may be enhanced in both a physical domain (e.g.,before application of a gamma curve) and in a perceptual domain (e.g.,after application of a gamma curve). Such a double enhancement mayimprove contrast and/or image quality. For example, a first enhancementmay be performed before application of a gamma curve (e.g., in aphysical space in which a pixel value is proportional to the photonsreceived by the photosite) by weighting a high frequency component imagemore heavily than a low frequency component image. For example, a secondenhancement may be performed after application of a gamma curve (e.g.,in a perceptual space in which a human eye may perceive the contrast) byweighting a high frequency component perceptual domain image moreheavily than a low frequency component perceptual domain image.

In some implementations, a single bilateral filter may be applied oncedetermine the low-frequency component images (e.g., base layers) in boththe physical domain and the perceptual domain. This may be accomplishedby using an approximation that the gamma curve mapping commutes with thebilateral filter operation. Application of a bilateral filter uses asignificant portion of the computing resources (e.g., memory andprocessor cycles) used for local tone mapping. Reusing the result of asingle application of the bilateral filter in both domains may yieldsubstantial saving in terms of implementation efficiency. In someimplementations, the low-frequency component image (e.g., a base layer)is determined by applying a bilateral filter before application of agamma curve and global tone mapping, then the low-frequency componentperceptual domain image is determined by applying the gamma curve andglobal tone mapping to the low-frequency component image.

To reduced computing resource consumption, a bilateral filter may beapplied at a lower resolution than the incoming image resolution. Forexample, the window of a bilateral filter used for determining alow-frequency component image (e.g., a base layer) may be a 300×300pixel block. A significant cost of implementing this bilateral filter isa 300 lines buffer used to have access to this window (e.g., a pixelneighborhood) during processing. To reduce this cost, a low resolutionwindow or neighborhood (e.g., one value for each 8×8 pixels) may beused, reducing this memory buffer size by a factor of 64. Candidatevalues for the bilateral filter may also be selected from this lowresolution image, which means that the number of candidates is alsodivided by 64, which also may reduce the number of processor cyclesused. This approach may facilitate implementation in an embeddedsolution. In some implementations, application of a bilateral filterincludes processing all candidate values up to a certain maximumdistance from a pixel under consideration (e.g., at the center of thewindow/neighborhood). In some implementations, similar results may beobtained by considering a smaller subset of these candidates (e.g.,around 30% of the candidates), reducing the consumption of computingresources. For example, all candidates up to a certain first distancemay be processed, then 50% of candidates between the first distance anda second distance, then 25% of candidates between the second distanceand a third distance.

Implementations are described in detail with reference to the drawings,which are provided as examples so as to enable those skilled in the artto practice the technology. The figures and examples are not meant tolimit the scope of the present disclosure to a single implementation orembodiment, and other implementations and embodiments are possible byway of interchange of, or combination with, some or all of the describedor illustrated elements. Wherever convenient, the same reference numberswill be used throughout the drawings to refer to same or like parts.

FIG. 1 is a diagram of an example of an image capture system 100 forcontent capture. As shown in FIG. 1, an image capture system 100 mayinclude an image capture apparatus 110, an external user interface (UI)device 120, or a combination thereof.

In some implementations, the image capture apparatus 110 may be amulti-face apparatus and may include multiple image capture devices,such as image capture devices 130, 132, 134 as shown in FIG. 1, arrangedin a structure 140, such as a cube-shaped cage as shown. Although threeimage capture devices 130, 132, 134 are shown for simplicity in FIG. 1,the image capture apparatus 110 may include any number of image capturedevices. For example, the image capture apparatus 110 shown in FIG. 1may include six cameras, which may include the three image capturedevices 130, 132, 134 shown and three cameras not shown.

In some implementations, the structure 140 may have dimensions, such asbetween 25 mm and 150 mm. For example, the length of each side of thestructure 140 may be 105 mm. The structure 140 may include a mountingport 142, which may be removably attachable to a supporting structure,such as a tripod, a photo stick, or any other camera mount (not shown).The structure 140 may be a rigid support structure, such that therelative orientation of the image capture devices 130, 132, 134 of theimage capture apparatus 110 may be maintained in relatively static orfixed alignment, except as described herein.

The image capture apparatus 110 may obtain, or capture, image content,such as images, video, or both, with a 360° field-of-view, which may bereferred to herein as panoramic or spherical content. For example, eachof the image capture devices 130, 132, 134 may include respectivelenses, for receiving and focusing light, and respective image sensorsfor converting the received and focused light to an image signal, suchas by measuring or sampling the light, and the multiple image capturedevices 130, 132, 134 may be arranged such that respective image sensorsand lenses capture a combined field-of-view characterized by a sphericalor near spherical field-of-view.

In some implementations, each of the image capture devices 130, 132, 134may have a respective field-of-view 170, 172, 174, such as afield-of-view 170, 172, 174 that 90° in a lateral dimension 180, 182,184 and includes 120° in a longitudinal dimension 190, 192, 194. In someimplementations, image capture devices 130, 132, 134 having overlappingfields-of-view 170, 172, 174, or the image sensors thereof, may beoriented at defined angles, such as at 90°, with respect to one another.In some implementations, the image sensor of the image capture device130 is directed along the X axis, the image sensor of the image capturedevice 132 is directed along the Y axis, and the image sensor of theimage capture device 134 is directed along the Z axis. The respectivefields-of-view 170, 172, 174 for adjacent image capture devices 130,132, 134 may be oriented to allow overlap for a stitching function. Forexample, the longitudinal dimension 190 of the field-of-view 170 for theimage capture device 130 may be oriented at 90° with respect to thelatitudinal dimension 184 of the field-of-view 174 for the image capturedevice 134, the latitudinal dimension 180 of the field-of-view 170 forthe image capture device 130 may be oriented at 90° with respect to thelongitudinal dimension 192 of the field-of-view 172 for the imagecapture device 132, and the latitudinal dimension 182 of thefield-of-view 172 for the image capture device 132 may be oriented at90° with respect to the longitudinal dimension 194 of the field-of-view174 for the image capture device 134.

The image capture apparatus 110 shown in FIG. 1 may have 420° angularcoverage in vertical and/or horizontal planes by the successive overlapof 90°, 120°, 90°, 120° respective fields-of-view 170, 172, 174 (not allshown) for four adjacent image capture devices 130, 132, 134 (not allshown). For example, fields-of-view 170, 172 for the image capturedevices 130, 132 and fields-of-view (not shown) for two image capturedevices (not shown) opposite the image capture devices 130, 132respectively may be combined to provide 420° angular coverage in ahorizontal plane. In some implementations, the overlap betweenfields-of-view of image capture devices 130, 132, 134 having a combinedfield-of-view including less than 360° angular coverage in a verticaland/or horizontal plane may be aligned and merged or combined to producea panoramic image. For example, the image capture apparatus 110 may bein motion, such as rotating, and source images captured by at least oneof the image capture devices 130, 132, 134 may be combined to form apanoramic image. As another example, the image capture apparatus 110 maybe stationary, and source images captured contemporaneously by eachimage capture device 130, 132, 134 may be combined to form a panoramicimage.

In some implementations, an image capture device 130, 132, 134 mayinclude a lens 150, 152, 154 or other optical element. An opticalelement may include one or more lens, macro lens, zoom lens,special-purpose lens, telephoto lens, prime lens, achromatic lens,apochromatic lens, process lens, wide-angle lens, ultra-wide-angle lens,fisheye lens, infrared lens, ultraviolet lens, perspective control lens,other lens, and/or other optical element. In some implementations, alens 150, 152, 154 may be a fisheye lens and produce fisheye, ornear-fisheye, field-of-view images. For example, the respective lenses150, 152, 154 of the image capture devices 130, 132, 134 may be fisheyelenses. In some implementations, images captured by two or more imagecapture devices 130, 132, 134 of the image capture apparatus 110 may becombined by stitching or merging fisheye projections of the capturedimages to produce an equirectangular planar image. For example, a firstfisheye image may be a round or elliptical image, and may be transformedto a first rectangular image, a second fisheye image may be a round orelliptical image, and may be transformed to a second rectangular image,and the first and second rectangular images may be arrangedside-by-side, which may include overlapping, and stitched together toform the equirectangular planar image.

Although not expressly shown in FIG. 1, in some implementations, each ofthe image capture devices 130, 132, 134 may include one or more imagesensors, such as a charge-coupled device (CCD) sensor, an active pixelsensor (APS), a complementary metal-oxide semiconductor (CMOS) sensor,an N-type metal-oxide-semiconductor (NMOS) sensor, and/or any otherimage sensor or combination of image sensors.

Although not expressly shown in FIG. 1, in some implementations, theimage capture apparatus 110 may include one or more microphones, whichmay receive, capture, and record audio information, which may beassociated with images acquired by the image sensors.

Although not expressly shown in FIG. 1, the image capture apparatus 110may include one or more other information sources or sensors, such as aninertial measurement unit (IMU), a global positioning system (GPS)receiver component, a pressure sensor, a temperature sensor, a heartrate sensor, or any other unit, or combination of units, that may beincluded in an image capture apparatus.

In some implementations, the image capture apparatus 110 may interfacewith or communicate with an external device, such as the external userinterface (UI) device 120, via a wired (not shown) or wireless (asshown) computing communication link 160. Although a single computingcommunication link 160 is shown in FIG. 1 for simplicity, any number ofcomputing communication links may be used. Although the computingcommunication link 160 shown in FIG. 1 is shown as a direct computingcommunication link, an indirect computing communication link, such as alink including another device or a network, such as the internet, may beused. In some implementations, the computing communication link 160 maybe a Wi-Fi link, an infrared link, a Bluetooth (BT) link, a cellularlink, a ZigBee link, a near field communications (NFC) link, such as anISO/IEC 23243 protocol link, an Advanced Network Technologyinteroperability (ANT+) link, and/or any other wireless communicationslink or combination of links. In some implementations, the computingcommunication link 160 may be an HDMI link, a USB link, a digital videointerface link, a display port interface link, such as a VideoElectronics Standards Association (VESA) digital display interface link,an Ethernet link, a Thunderbolt link, and/or other wired computingcommunication link.

In some implementations, the user interface device 120 may be acomputing device, such as a smartphone, a tablet computer, a phablet, asmart watch, a portable computer, and/or another device or combinationof devices configured to receive user input, communicate informationwith the image capture apparatus 110 via the computing communicationlink 160, or receive user input and communicate information with theimage capture apparatus 110 via the computing communication link 160.

In some implementations, the image capture apparatus 110 may transmitimages, such as panoramic images, or portions thereof, to the userinterface device 120 via the computing communication link 160, and theuser interface device 120 may store, process, display, or a combinationthereof the panoramic images.

In some implementations, the user interface device 120 may display, orotherwise present, content, such as images or video, acquired by theimage capture apparatus 110. For example, a display of the userinterface device 120 may be a viewport into the three-dimensional spacerepresented by the panoramic images or video captured or created by theimage capture apparatus 110.

In some implementations, the user interface device 120 may communicateinformation, such as metadata, to the image capture apparatus 110. Forexample, the user interface device 120 may send orientation informationof the user interface device 120 with respect to a defined coordinatesystem to the image capture apparatus 110, such that the image captureapparatus 110 may determine an orientation of the user interface device120 relative to the image capture apparatus 110. Based on the determinedorientation, the image capture apparatus 110 may identify a portion ofthe panoramic images or video captured by the image capture apparatus110 for the image capture apparatus 110 to send to the user interfacedevice 120 for presentation as the viewport. In some implementations,based on the determined orientation, the image capture apparatus 110 maydetermine the location of the user interface device 120 and/or thedimensions for viewing of a portion of the panoramic images or video.

In an example, a user may rotate (sweep) the user interface device 120through an arc or path 122 in space, as indicated by the arrow shown at122 in FIG. 1. The user interface device 120 may communicate displayorientation information to the image capture apparatus 110 using acommunication interface such as the computing communication link 160.The image capture apparatus 110 may provide an encoded bitstream toenable viewing of a portion of the panoramic content corresponding to aportion of the environment of the display location as the image captureapparatus 110 traverses the path 122. Accordingly, display orientationinformation from the user interface device 120 may be transmitted to theimage capture apparatus 110 to control user selectable viewing ofcaptured images and/or video.

In some implementations, the image capture apparatus 110 may communicatewith one or more other external devices (not shown) via wired orwireless computing communication links (not shown).

In some implementations, data, such as image data, audio data, and/orother data, obtained by the image capture apparatus 110 may beincorporated into a combined multimedia stream. For example, themultimedia stream may include a video track and/or an audio track. Asanother example, information from various metadata sensors and/orsources within and/or coupled to the image capture apparatus 110 may beprocessed to produce a metadata track associated with the video and/oraudio track. The metadata track may include metadata, such as whitebalance metadata, image sensor gain metadata, sensor temperaturemetadata, exposure time metadata, lens aperture metadata, bracketingconfiguration metadata and/or other parameters. In some implementations,a multiplexed stream may be generated to incorporate a video and/oraudio track and one or more metadata tracks.

In some implementations, the user interface device 120 may implement orexecute one or more applications, such as GoPro Studio, GoPro App, orboth, to manage or control the image capture apparatus 110. For example,the user interface device 120 may include an application for controllingcamera configuration, video acquisition, video display, or any otherconfigurable or controllable aspect of the image capture apparatus 110.

In some implementations, the user interface device 120, such as via anapplication (e.g., GoPro App), may generate and share, such as via acloud-based or social media service, one or more images, or short videoclips, such as in response to user input.

In some implementations, the user interface device 120, such as via anapplication (e.g., GoPro App), may remotely control the image captureapparatus 110, such as in response to user input.

In some implementations, the user interface device 120, such as via anapplication (e.g., GoPro App), may display unprocessed or minimallyprocessed images or video captured by the image capture apparatus 110contemporaneously with capturing the images or video by the imagecapture apparatus 110, such as for shot framing, which may be referredto herein as a live preview, and which may be performed in response touser input.

In some implementations, the user interface device 120, such as via anapplication (e.g., GoPro App), may mark one or more key momentscontemporaneously with capturing the images or video by the imagecapture apparatus 110, such as with a HiLight Tag, such as in responseto user input.

In some implementations, the user interface device 120, such as via anapplication (e.g., GoPro App), may display, or otherwise present, marksor tags associated with images or video, such as HiLight Tags, such asin response to user input. For example, marks may be presented in aGoPro Camera Roll application for location review and/or playback ofvideo highlights.

In some implementations, the user interface device 120, such as via anapplication (e.g., GoPro App), may wirelessly control camera software,hardware, or both. For example, the user interface device 120 mayinclude a web-based graphical interface accessible by a user forselecting a live or previously recorded video stream from the imagecapture apparatus 110 for display on the user interface device 120.

In some implementations, the user interface device 120 may receiveinformation indicating a user setting, such as an image resolutionsetting (e.g., 3840 pixels by 2160 pixels), a frame rate setting (e.g.,60 frames per second (fps)), a location setting, and/or a contextsetting, which may indicate an activity, such as mountain biking, inresponse to user input, and may communicate the settings, or relatedinformation, to the image capture apparatus 110.

FIG. 2A is a block diagram of an example of a system 200 configured forimage capture and tone mapping. The system 200 includes an image capturedevice 210 (e.g., a camera or a drone) that includes a processingapparatus 212 that is configured to receive a first image from the firstimage sensor 214 and receive a second image from the second image sensor216. The processing apparatus 212 may be configured to perform imagesignal processing (e.g., filtering, tone mapping, stitching, and/orencoding) to generate output images based on image data from the imagesensors 214 and 216. The image capture device 210 includes acommunications interface 218 for transferring images to other devices.The image capture device 210 includes a user interface 220, which mayallow a user to control image capture functions and/or view images. Theimage capture device 210 includes a battery 222 for powering the imagecapture device 210. The components of the image capture device 210 maycommunicate with each other via the bus 224. The system 200 may be usedto implement techniques described in this disclosure, such as thetechnique 300 of FIG. 3, the technique 400 of FIG. 4, or the technique500 of FIG. 5.

The processing apparatus 212 may include one or more processors havingsingle or multiple processing cores. The processing apparatus 212 mayinclude memory, such as random access memory device (RAM), flash memory,or any other suitable type of storage device such as a non-transitorycomputer readable memory. The memory of the processing apparatus 212 mayinclude executable instructions and data that can be accessed by one ormore processors of the processing apparatus 212. For example, theprocessing apparatus 212 may include one or more DRAM modules such asdouble data rate synchronous dynamic random-access memory (DDR SDRAM).In some implementations, the processing apparatus 212 may include adigital signal processor (DSP). In some implementations, the processingapparatus 212 may include an application specific integrated circuit(ASIC). For example, the processing apparatus 212 may include a customimage signal processor.

The first image sensor 214 and the second image sensor 216 areconfigured to detect light of a certain spectrum (e.g., the visiblespectrum or the infrared spectrum) and convey information constitutingan image as electrical signals (e.g., analog or digital signals). Forexample, the image sensors 214 and 216 may include charge-coupleddevices (CCD) or active pixel sensors in complementarymetal-oxide-semiconductor (CMOS). The image sensors 214 and 216 maydetect light incident through respective lens (e.g., a fisheye lens). Insome implementations, the image sensors 214 and 216 include digital toanalog converters. In some implementations, the image sensors 214 and216 are held in a fixed orientation with respective fields of view thatoverlap.

The image capture device 210 may include a communications interface 218,which may enable communications with a personal computing device (e.g.,a smartphone, a tablet, a laptop computer, or a desktop computer). Forexample, the communications interface 218 may be used to receivecommands controlling image capture and processing in the image capturedevice 210. For example, the communications interface 218 may be used totransfer image data to a personal computing device. For example, thecommunications interface 218 may include a wired interface, such as ahigh-definition multimedia interface (HDMI), a universal serial bus(USB) interface, or a FireWire interface. For example, thecommunications interface 218 may include a wireless interface, such as aBluetooth interface, a ZigBee interface, and/or a Wi-Fi interface.

The image capture device 210 may include a user interface 220. Forexample, the user interface 220 may include an LCD display forpresenting images and/or messages to a user. For example, the userinterface 220 may include a button or switch enabling a person tomanually turn the image capture device 210 on and off. For example, theuser interface 220 may include a shutter button for snapping pictures.

The image capture device 210 may include a battery 222 that powers theimage capture device 210 and/or its peripherals. For example, thebattery 222 may be charged wirelessly or through a micro-USB interface.

FIG. 2B is a block diagram of an example of a system 230 configured forimage capture and tone mapping. The system 230 includes an image capturedevice 240 and a personal computing device 260 that communicate via acommunications link 250. The image capture device 240 includes a firstimage sensor 242 and a second image sensor 244 that are configured tocapture respective images. The image capture device 240 includes acommunications interface 246 configured to transfer images via thecommunication link 250 to the personal computing device 260. Thepersonal computing device 260 includes a processing apparatus 262 thatis configured to receive, using the communications interface 266, afirst image from the first image sensor, and receive a second image fromthe second image sensor 244. The processing apparatus 262 may beconfigured to perform image signal processing (e.g., filtering, tonemapping, stitching, and/or encoding) to generate output images based onimage data from the image sensors 242 and 244. The system 230 may beused to implement techniques described in this disclosure, such as thetechnique 300 of FIG. 3, the technique 400 of FIG. 4, or the technique500 of FIG. 5.

The first image sensor 242 and the second image sensor 244 areconfigured to detect light of a certain spectrum (e.g., the visiblespectrum or the infrared spectrum) and convey information constitutingan image as electrical signals (e.g., analog or digital signals). Forexample, the image sensors 242 and 244 may include charge-coupleddevices (CCD) or active pixel sensors in complementarymetal-oxide-semiconductor (CMOS). The image sensors 242 and 244 maydetect light incident through respective lens (e.g., a fisheye lens). Insome implementations, the image sensors 242 and 244 include digital toanalog converters. In some implementations, the image sensors 242 and244 are held in a fixed relative orientation with respective fields ofview that overlap. Image signals from the image sensors 242 and 244 maybe passed to other components of the image capture device 240 via thebus 248.

The communications link 250 may be a wired communications link or awireless communications link. The communications interface 246 and thecommunications interface 266 may enable communications over thecommunications link 250. For example, the communications interface 246and the communications interface 266 may include a high-definitionmultimedia interface (HDMI), a universal serial bus (USB) interface, aFireWire interface, a Bluetooth interface, a ZigBee interface, and/or aWi-Fi interface. For example, the communications interface 246 and thecommunications interface 266 may be used to transfer image data from theimage capture device 240 to the personal computing device 260 for imagesignal processing (e.g., filtering, tone mapping, stitching, and/orencoding) to generate output images based on image data from the imagesensors 242 and 244.

The processing apparatus 262 may include one or more processors havingsingle or multiple processing cores. The processing apparatus 262 mayinclude memory, such as random access memory device (RAM), flash memory,or any other suitable type of storage device such as a non-transitorycomputer readable memory. The memory of the processing apparatus 262 mayinclude executable instructions and data that can be accessed by one ormore processors of the processing apparatus 262. For example, theprocessing apparatus 262 may include one or more DRAM modules such asdouble data rate synchronous dynamic random-access memory (DDR SDRAM).In some implementations, the processing apparatus 262 may include adigital signal processor (DSP). In some implementations, the processingapparatus 262 may include an application specific integrated circuit(ASIC). For example, the processing apparatus 262 may include a customimage signal processor. The processing apparatus 262 may exchange data(e.g., image data) with other components of the personal computingdevice 260 via the bus 268.

The personal computing device 260 may include a user interface 264. Forexample, the user interface 264 may include a touchscreen display forpresenting images and/or messages to a user and receiving commands froma user. For example, the user interface 264 may include a button orswitch enabling a person to manually turn the personal computing device260 on and off. In some implementations, commands (e.g., start recordingvideo, stop recording video, or snap photograph) received via the userinterface 264 may be passed on to the image capture device 240 via thecommunications link 250.

Global tone mapping can be applied as a variable gain that is applied onthe linear RGB vales according to their luminance in order to have abetter repartition of the information on the output range. This gain maydepend on the input histogram of luminance values and a target histogramthat has to be matched (e.g., a flat histogram to equalize the image ora Gaussian histogram to have a better enhancement ofshadows/highlights). Consider a pixel value x_(n)=[R,G,B]{circumflexover ( )}T. A global tone mapping gain λ( ) may be applied as follows:{acute over (x)}_(n)=λ(Y(x_(n)))*x_(n) where {acute over (x)}_(n) is aglobal tone mapped pixel value and Y(x) is a discrete approximation ofthe luminance defined by a linear combination of the R, G and Bchannels.

Global tone mapping can be a good approach to increase the entropy or tomatch a given histogram of pixel luminance but doesn't take into accountthe spatial repartition of the image data. Indeed, two images can havethe exact same histogram but can represent either a smooth gradient or anoisy image. A goal of the global tone mapping is to widen the ranges ofthe input dynamic range that represent more information of the image atthe expense of a compression in the range(s) of luminance values of thatrepresent less information. This leads to a loss of contrast in someareas of the image. A resulting loss of contrast may not be thatimportant if the compressed information is not gathered at the samelocation in the image, but when compressed information is spatiallyconcentrated, it can lead to unnatural and/or low quality rendering ofthe image. In order to preserve or enhance some of the details that maybe lost using global tone mapping only, spatial information may beintroduced, which may enable keeping the contrast in these areas. Alocal tone mapping may help to reach this goal.

For example, a process for local tone mapping may include: separatingthe low and high frequencies of the input image in order to preserve thehigh frequency contrast and to compress low frequency transitions. Someinexpensive (in terms of computing resources) local tone mappingapproaches are based on unsharp mask techniques, which may introducehalos. In order to have strong local tone mapping on high dynamic rangeimages, halos may be suppressed by using an edge-aware filtering (e.g.,a bilateral filter). High and low frequency components of an image maythen be recombined to achieve preservation and/or amplification of thedetails.

FIG. 3 is a flowchart of an example technique 300 for local tone mappingof a captured image. The technique 300 includes receiving 310 the imagefrom an image sensor; applying 320 a filter to the image to obtain alow-frequency component image and a high-frequency component image;applying 330 a non-linear mapping to the low-frequency component imageto obtain gains for respective image portions; applying 340 the gainsfor respective image portions to corresponding image portions of theimage to obtain an enhanced image; and storing, displaying, ortransmitting 370 an output image based on the enhanced image. Forexample, the technique 300 may be implemented by the system 200 of FIG.2A or the system 230 of FIG. 2B. For example, the technique 300 may beimplemented by an image capture device, such the image capture device210 shown in FIG. 2, or an image capture apparatus, such as the imagecapture apparatus 110 shown in FIG. 1. For example, the technique 300may be implemented by a personal computing device, such as the personalcomputing device 260.

The technique 300 includes receiving 310 the image from the imagesensor. The image sensor may be part of an image capture apparatus(e.g., the image capture apparatus 110, the image capture device 210, orthe image capture device 240). In some implementations, the image sensormay be attached to a processing apparatus that implements the technique300. For example, the image may be received 310 from the image sensorvia a bus (e.g., the bus 224). In some implementations, the image may bereceived 310 via a communications link (e.g., the communications link250). For example, the image may be received 310 via a wireless or wiredcommunications interface (e.g., Wi-Fi, Bluetooth, USB, HDMI, WirelessUSB, Near Field Communication (NFC), Ethernet, a radio frequencytransceiver, and/or other interfaces). For example, the image may bereceived 310 via communications interface 266. For example, the imagemay be received 310 as an input image signal, which may represent eachpixel value in a defined format, such as in a RAW image format. In someimplementations, the image may be frame of video, i.e., one of asequence of images of a video. In some implementations, the image isreceived 310 directly from the image sensor without intermediate imageprocessing. In some implementations, the image is received 310 afterbeing subjected to intermediate image processing (e.g., correction ofdead pixels, band processing, decoupling of vertical blanking, spatialnoise reduction, and/or temporal noise reduction).

The technique 300 includes applying 320 a filter to the image to obtaina low-frequency component image and a high-frequency component image.The filter may include a low-pass filter. In some implementations, thefilter may include a Guassian blur. In some implementations, the filtermay include a bilateral filter. For example, a bilateral filter may bedefined as follows. Consider a pixel value x_(n) of a pixel at positionp_(n) from the image in a linear space RGB format (e.g., a luminancevalue Y(x_(n)) may be determined as a linear combination of RGB channelvalues for the pixel). A cross-bilateral filter may be applied on xguided by γ(Y(x)) where γ( ) is the gamma curve that will later beapplied on the image. A goal of this gamma curve is to be able to filterthe contrast in a perceptual space.

$\begin{matrix}{{{\overset{\_}{x}}_{n} = {\frac{1}{N\_\Omega}{\sum\limits_{p_{k}\mspace{14mu}{in}\mspace{14mu}{\Omega{(p_{n})}}}\left\lbrack {x_{k}\mspace{14mu}{w\_ s}\left( {p_{k},p_{n}} \right){w\_ r}\left( {{\gamma\left( {Y\left( x_{k} \right)} \right)},{\gamma\left( {Y\left( x_{n} \right)} \right)}} \right)} \right\rbrack}}}\mspace{70mu}{{N\_\Omega} = {\sum\limits_{p_{k}\mspace{14mu}{in}\mspace{14mu}{\Omega{(p_{n})}}}\left\lbrack {{w\_ s}\left( {p_{k},p_{n}} \right){w\_ r}\left( {{\gamma\left( {Y\left( x_{k} \right)} \right)},{\gamma\left( {Y\left( x_{n} \right)} \right)}} \right)} \right\rbrack}}} & {{Equation}\mspace{14mu} 1}\end{matrix}$

where x is a low-frequency component image (e.g., a base layer);Ω(p_(n)) is a window of pixels around the position p_(n); w_s( ) is aspatial Gaussian weighting function centered on p_(n) and specified byσ_s (e.g., σ_s may be chosen to be approximately 50 for a 12 mega-pixelimage and a radius of r=3*σ_s may be used for the spatial Gaussianweighting function); and w_r( ) is a similarity function based on thedifference of intensity between γ(Y(x_(k)) and γ(Y(x₀)) defined as:

${{w\_ r}\left( {u,v} \right)} = {{\frac{1}{ɛ^{\rho}}\mspace{14mu}{if}\mspace{14mu}{{u - v}}} < ɛ}$${{w\_ r}\left( {u,v} \right)} = {\frac{1}{{{u - v}}^{\rho}}\mspace{14mu}{otherwise}}$For example, the constant ρ may be set to 1.5 the constant ε may be setto 0.05. In some implementations, the high-frequency component image isdetermined as a compliment of the low-frequency component image (e.g.,as (x−x)).

In some implementations, the computing resources (e.g., memory and/orprocessor cycles) consumed to apply 320 the filter may be reduced usingapproximations or other techniques to reduce complexity. For example,where the filter includes a bilateral filter, a reduced resolution image(e.g., a bin 8 image) may be determined based on the image that is at alower resolution than the image, and applying the bilateral filter mayinclude processing pixels of the reduced resolution image as candidates(i.e., pixels within the window Ω(p_(n)) of the bilateral filter). Forexample, where a bin 8 image is used for the bilateral filter ratherthan a full resolution image the number of candidates processed by thebilateral filter may be reduced by a factor of approximately 64. In someimplementations, an anti-alias Guassian filter (e.g., σ=8 pixels) may beapplied prior to subsampling to determine the reduced resolution image.Applying 320 a bilateral filter may include subsampling candidateswithin a range of distances from a kernel center. The subsampling ofcandidates may be implemented with a sparse kernel or window Ω(p_(n))for the bilateral filter. For example, applying 320 a bilateral filtermay include subsampling candidates at a first subsampling factor withina first range of distances from a kernel center, and subsamplingcandidates at a second subsampling factor within a second range ofdistances from the kernel center. Filter weights of a sparse bilateralfilter may be modified in order to mitigate changes in the filteringstrength relative to a full resolution bilateral filter (e.g., thebilateral filter of Equation 1). An interpolation may be used tointerpolate the candidates according to the position of the currentpixel and to have a smooth continuous output.

In order to preserve details, a goal may be to apply the same gain toall pixels belonging to the same object. Toward this end, a global tonemapping gain may be driven by the low-frequency component image: {acuteover (x)}_(n)=λ(x _(n))x_(n), which can be rewritten as {acute over(x)}_(n)=λ(x _(n))(x _(n)+(x_(n)−x _(n))). From there, it can be deducedthat the values of the low frequencies and high frequencies of theoutput image respectively are

=λ(x _(n))x _(n) and {acute over (x)}_(n)−

=λ(x _(n))(x_(n)−x _(n)). In this expression, we can see that the gainis driven by the low-frequency component image (e.g., a local mean ofthe image pixel values), but is applied both to the low frequencies x_(n) and to the high frequencies (x_(n)−x _(n)). Therefore the detailsmay be enhanced the same ways as the mean. This approach may serve topreserve local contrast and preserve the details that would have beeneither amplified or compressed if the global tone mapping was appliedwithout modification to account for local variation.

The technique 300 includes applying 330 a non-linear mapping to thelow-frequency component image to obtain gains for respective imageportions (e.g., pixels or blocks of pixels). For example, the tone curveλ( ) may be applied to the low-frequency component image to determinegains for respective image portions. In some implementations, thenon-linear mapping is determined based on a histogram analysis of imageportions of the of the low-frequency component image. For example, thetone curve λ( ) may be determined based on a histogram of values in thelow-frequency component image x. For example, the tone curve λ( ) may bedetermined to achieve a target histogram or distribution of luminancevalues for a resulting output image (e.g., to equalize the histogram ofthe low-frequency component image).

The technique 300 includes applying 340 the gains for respective imageportions (e.g., pixels or blocks of pixels) to corresponding imageportions of the image to obtain an enhanced image. The gains may beapplied 340 by multiplying the gains with corresponding pixel values ofthe image. For example, the obtained gains λ(x _(n)) may be applied 340according to {acute over (x)}_(n)=λ(x _(n))x_(n), where {acute over (x)}is a tone mapped enhanced image.

The technique 300 includes storing, displaying, or transmitting 370 anoutput image based on the enhanced image. In some implementations, theoutput image is the enhanced image. In some implementations, theenhanced image may by subject to additional image processing (e.g.,perceptual tone mapping with a gamma curve γ( ) lens distortioncorrection, electronic rolling shutter correction, stitching withparallax correction and blending to combine images from multiple imagesensors, electronic image stabilization, and/or output projection) todetermine the output image. For example, the output image may betransmitted 370 to an external device (e.g., a personal computingdevice) for display or storage. For example, the output image may bestored 370 in memory of a processing apparatus (e.g., the processingapparatus 212 or the processing apparatus 262). For example, the outputimage may be displayed 370 in the user interface 220 or in the userinterface 264. For example, the output image may be transmitted 370 viathe communications interface 218.

Once the separation between the high and low frequencies in an image hasbeen performed, it is possible to take advantage of this not only topreserve the contrast but also to perform extra contrast or detailenhancement. Tone mapping operations may be applied in differentdomains: (1) to an image represented in a physical domain, once we haveall the information about the real color of the scene; or (2) to animage represented in a perceptual domain, after a gamma curve γ( ) hasbeen applied to compensate for the eye contrast response non-uniformity.In the physical domain a pixel value may be proportional to the numberof photons received by the sensor. Local contrast enhancement in thephysical domain may lead to more natural and plausible scenes but maylack contrast in the highlights due to the compression performed by whena gamma curve is later applied. In the perceptual domain however, thesame amplification may be performed on both the shadows and thehighlights independently of the gamma curve compression. When welltuned, local contrast enhancement in the perceptual domain can lead topunchier images but may look more like an unnatural high dynamic rangecontrast enhancement when stronger. To balance these concerns, localtone mapping operations may be applied in both the physical domain andin the perceptual domain.

FIG. 4 is a flowchart of an example technique 400 for local tone mappingof a captured image. The technique 400 includes receiving 410 an imagefrom an image sensor; applying 420 a filter to the image to obtain alow-frequency component image and a high-frequency component image;determining 430 a first enhanced image based on a weighted sum of thelow-frequency component image and the high-frequency component image,where the high-frequency component image is weighted more than thelow-frequency component image; determining 440 a second enhanced imagebased on the first enhanced image and a tone mapping; and storing,displaying, or transmitting 450 an output image based on the secondenhanced image. For example, the technique 400 may be implemented by thesystem 200 of FIG. 2A or the system 230 of FIG. 2B. For example, thetechnique 400 may be implemented by an image capture device, such theimage capture device 210 shown in FIG. 2, or an image capture apparatus,such as the image capture apparatus 110 shown in FIG. 1. For example,the technique 400 may be implemented by a personal computing device,such as the personal computing device 260.

The technique 400 includes receiving 410 an image from an image sensor.The image sensor may be part of an image capture apparatus (e.g., theimage capture apparatus 110, the image capture device 210, or the imagecapture device 240). In some implementations, the image sensor may beattached to a processing apparatus that implements the technique 400.For example, the image may be received 410 from the image sensor via abus (e.g., the bus 224). In some implementations, the image may bereceived 410 via a communications link (e.g., the communications link250). For example, the image may be received 410 via a wireless or wiredcommunications interface (e.g., Wi-Fi, Bluetooth, USB, HDMI, WirelessUSB, Near Field Communication (NFC), Ethernet, a radio frequencytransceiver, and/or other interfaces). For example, the image may bereceived 410 via communications interface 266. For example, the imagemay be received 410 as an input image signal, which may represent eachpixel value in a defined format, such as in a RAW image format. In someimplementations, the image may be frame of video, i.e., one of asequence of images of a video. In some implementations, the image isreceived 410 directly from the image sensor without intermediate imageprocessing. In some implementations, the image is received 410 afterbeing subjected to intermediate image processing (e.g., correction ofdead pixels, band processing, decoupling of vertical blanking, spatialnoise reduction, and/or temporal noise reduction).

The technique 400 includes applying 420 a filter to the image to obtaina low-frequency component image and a high-frequency component image.The filter may include a low-pass filter. In some implementations, thefilter may include a Guassian blur. In some implementations, the filtermay include a bilateral filter. For example, the bilateral filter may bedefined by Equation 1 above. In some implementations, the high-frequencycomponent image is determined as a compliment of the low-frequencycomponent image (e.g., as (x−x)).

The technique 400 includes determining 430 a first enhanced image basedon a weighted sum of the low-frequency component image and thehigh-frequency component image, where the high-frequency component imageis weighted more than the low-frequency component image. Local contrastenhancement may be performed by tuning the proportion of details in anenhanced image that is determined as weighted sum of component images,including the low-frequency component image and the high-frequencycomponent image. For example, the first enhanced image may be determinedaccording to:{acute over (x)} _(n) =x _(n)α(x _(n) −x _(n))where {acute over (x)} is the first enhanced image, x is an imagerepresented in a physical domain, and α>1 is a weight chosen to enhancedetails by weighting the high-frequency component image more heavily. Insome implementations, determining 430 the first enhanced image mayinclude checking for overflow and/or underflow conditions. For example,the first enhanced image may be determined 430 using the technique 600of FIG. 6.

The technique 400 includes determining 440 a second enhanced image basedon the first enhanced image and a tone mapping. In some implementations,a global tone mapping, based on a tone curve λ( ), may be applied to thefirst enhanced image to determine 440 the second enhanced image. Forexample, the second enhanced image

may be determined 440 according to:

=λ({acute over (x)} _(n)){acute over (x)} _(n)In some implementations, a tone mapping that depends on pixels in alocal area may be applied to the first enhanced image to determine 440the second enhanced image. For example, determining the second enhancedimage based on the first enhanced image and the tone mapping may includeapplying the tone mapping to the low-frequency component image to obtaingains for respective image portions; and applying the gains forrespective image portions to corresponding image portions of the firstenhanced image. For example, the second enhanced image

may be determined 440 according to:

=λ( x _(n)){acute over (x)} _(n)For example, the tone curve λ( ) may be determined based on a histogramof values in the low-frequency component image x. For example, the tonecurve λ( ) may be determined to achieve a target histogram ordistribution of luminance values for a resulting output image (e.g., toequalize the histogram of the low-frequency component image).

The technique 400 includes storing, displaying, or transmitting 450 anoutput image based on the second enhanced image. In someimplementations, the output image is the second d enhanced image. Insome implementations, the second enhanced image may by subject toadditional image processing (e.g., perceptual tone mapping with a gammacurve γ( ), lens distortion correction, electronic rolling shuttercorrection, stitching with parallax correction and blending to combineimages from multiple image sensors, electronic image stabilization,and/or output projection) to determine the output image. For example,the output image may be transmitted 450 to an external device (e.g., apersonal computing device) for display or storage. For example, theoutput image may be stored 450 in memory of a processing apparatus(e.g., the processing apparatus 212 or the processing apparatus 262).For example, the output image may be displayed 450 in the user interface220 or in the user interface 264. For example, the output image may betransmitted 450 via the communications interface 218.

FIG. 5 is a flowchart of an example technique 500 for local tone mappingof a captured image. The technique 500 includes receiving 510 the imagefrom the image sensor; applying 520 a filter to the image to obtain alow-frequency component image and a high-frequency component image;determining 530 a first enhanced image based on a weighted sum of thelow-frequency component image and the high-frequency component image,where the high-frequency component image is weighted more than thelow-frequency component image; determining 540 a second enhanced imagebased on the first enhanced image and a tone mapping; determining 550 aperceptual domain image based on the second enhanced image and a gammacurve that models human perception of contrast; determining 560 alow-frequency component perceptual domain image and a high-frequencycomponent perceptual domain image as components of the perceptual domainimage; determining 570 a third enhanced image based on a weighted sum ofthe low-frequency component perceptual domain image and thehigh-frequency component perceptual domain image, where thehigh-frequency component perceptual domain image is weighted more thanthe low-frequency component perceptual domain image; and storing,displaying, or transmitting 580 an output image based on the thirdenhanced image. For example, the technique 500 may be implemented by thesystem 200 of FIG. 2A or the system 230 of FIG. 2B. For example, thetechnique 500 may be implemented by an image capture device, such theimage capture device 210 shown in FIG. 2, or an image capture apparatus,such as the image capture apparatus 110 shown in FIG. 1. For example,the technique 500 may be implemented by a personal computing device,such as the personal computing device 260.

The technique 500 includes receiving 510 an image from an image sensor.The image sensor may be part of an image capture apparatus (e.g., theimage capture apparatus 110, the image capture device 210, or the imagecapture device 240). In some implementations, the image sensor may beattached to a processing apparatus that implements the technique 500.For example, the image may be received 510 from the image sensor via abus (e.g., the bus 224). In some implementations, the image may bereceived 510 via a communications link (e.g., the communications link250). For example, the image may be received 510 via a wireless or wiredcommunications interface (e.g., Wi-Fi, Bluetooth, USB, HDMI, WirelessUSB, Near Field Communication (NFC), Ethernet, a radio frequencytransceiver, and/or other interfaces). For example, the image may bereceived 510 via communications interface 266. For example, the imagemay be received 510 as an input image signal, which may represent eachpixel value in a defined format, such as in a RAW image format. In someimplementations, the image may be frame of video, i.e., one of asequence of images of a video. In some implementations, the image isreceived 510 directly from the image sensor without intermediate imageprocessing. In some implementations, the image is received 510 afterbeing subjected to intermediate image processing (e.g., correction ofdead pixels, band processing, decoupling of vertical blanking, spatialnoise reduction, and/or temporal noise reduction).

The technique 500 includes applying 520 a filter to the image to obtaina low-frequency component image and a high-frequency component image.The filter may include a low-pass filter. In some implementations, thefilter may include a Guassian blur. In some implementations, the filtermay include a bilateral filter. For example, the bilateral filter may bedefined by Equation 1 above. In some implementations, the high-frequencycomponent image is determined as a compliment of the low-frequencycomponent image (e.g., as (x−x)).

In some implementations, the computing resources (e.g., memory and/orprocessor cycles) consumed to apply 520 the filter may be reduced usingapproximations or other techniques to reduce complexity. For example,where the filter includes a bilateral filter, a reduced resolution image(e.g., a bin 8 image) may be determined based on the image that is at alower resolution than the image, and applying the bilateral filter mayinclude processing pixels of the reduced resolution image as candidates(i.e., pixels within the window Ω(p_(n)) of the bilateral filter). Forexample, where a bin 8 image is used for the bilateral filter ratherthan a full resolution image the number of candidates processed by thebilateral filter may be reduced by a factor of approximately 64. In someimplementations, an anti-alias Guassian filter (e.g., σ=8 pixels) may beapplied prior to subsampling to determine the reduced resolution image.Applying 520 a bilateral filter may include subsampling candidateswithin a range of distances from a kernel center. The subsampling ofcandidates may be implemented with a sparse kernel or window Ω(p_(n))for the bilateral filter. For example, applying 520 a bilateral filtermay include subsampling candidates at a first subsampling factor withina first range of distances from a kernel center, and subsamplingcandidates at a second subsampling factor within a second range ofdistances from the kernel center. Filter weights of a sparse bilateralfilter may be modified in order to mitigate changes in the filteringstrength relative to a full resolution bilateral filter (e.g., thebilateral filter of Equation 1). An interpolation may be used tointerpolate the candidates according to the position of the currentpixel and to have a smooth continuous output.

The technique 500 includes determining 530 a first enhanced image basedon a weighted sum of the low-frequency component image and thehigh-frequency component image, where the high-frequency component imageis weighted more than the low-frequency component image. Local contrastenhancement may be performed by tuning the proportion of details in anenhanced image that is determined as weighted sum of component images,including the low-frequency component image and the high-frequencycomponent image. For example, the first enhanced image may be determinedaccording to:{acute over (x)} _(n) =x _(n)+α(x _(n) −x _(n))where {acute over (x)} is the first enhanced image, x is an imagerepresented in a physical domain, and α>1 is a weight chosen to enhancedetails by weighting the high-frequency component image more heavily. Insome implementations, determining 530 the first enhanced image mayinclude checking for overflow and/or underflow conditions. For example,the first enhanced image may be determined 530 using the technique 600of FIG. 6.

The technique 500 includes determining 540 a second enhanced image basedon the first enhanced image and a tone mapping. In some implementations,a global tone mapping, based on a tone curve λ( ), may be applied to thefirst enhanced image to determine 540 the second enhanced image. Forexample, the second enhanced image

may be determined 540 according to:

=λ({acute over (x)} _(n)){acute over (x)} _(n)In some implementations, a tone mapping that depends on pixels in alocal area may be applied to the first enhanced image to determine 540the second enhanced image. For example, determining the second enhancedimage based on the first enhanced image and the tone mapping may includeapplying the tone mapping to the low-frequency component image to obtaingains for respective image portions; and applying the gains forrespective image portions to corresponding image portions of the firstenhanced image. For example, the second enhanced image

may be determined 540 according to:

=λ( x _(n)){acute over (x)} _(n)In some implementations, the tone mapping may be determined based on ahistogram analysis of image portions of the of the low-frequencycomponent image. For example, the tone curve λ( ) may be determinedbased on a histogram of values in the low-frequency component image

. For example, the tone curve λ( ) may be determined to achieve a targethistogram or distribution of luminance values for a resulting outputimage (e.g., to equalize the histogram of the low-frequency componentimage).

The technique 500 includes determining 550 a perceptual domain imagebased on the second enhanced image and a gamma curve that models humanperception of contrast. The perceptual domain image y may be determined550 as y_(n)=γ(

).

The technique 500 includes determining 560 a low-frequency componentperceptual domain image and a high-frequency component perceptual domainimage as components of the perceptual domain image. In someimplementations, the low-frequency component perceptual domain image ymay be determined 560 by applying a filter (e.g., a bilateral filter ora Guassian blur) to the perceptual domain image y. However applying afilter a second time in the perceptual domain can consume significantcomputing resources (e.g., memory and/or processor cycles). To conservecomputing resources, in some implementations, the low-frequencycomponent perceptual domain image is determined 560 using anapproximation that it is equal to the result of applying the gamma curveto the low-frequency component image that was previously determined inthe physical domain, thus avoiding a second resource intensiveapplication of a filter (e.g., a bilateral filter). Determining 560 thelow-frequency component perceptual domain image may include applying atransformation, based on the gamma curve, to a result of applying thetone mapping to the low-frequency component image. For example, thelow-frequency component perceptual domain image y may be determined 560according to: y _(n)=γ(λ(x _(n))x _(n)). For example, the high-frequencycomponent perceptual domain image may be determined 560 as a complimentof the low-frequency component perceptual domain image, i.e., as (y−y).

The technique 500 includes determining 570 a third enhanced image basedon a weighted sum of the low-frequency component perceptual domain imageand the high-frequency component perceptual domain image, where thehigh-frequency component perceptual domain image is weighted more thanthe low-frequency component perceptual domain image. Local contrastenhancement may be performed by tuning the proportion of details in anenhanced perceptual domain image that is determined as weighted sum ofcomponent images, including the low-frequency component perceptualdomain image and the high-frequency component perceptual domain image.For example, the third enhanced image may be determined according to:ý _(n) =y _(n)+β(y _(n) −y _(n))where ý is the third enhanced image, y is the perceptual domain image,and β>1 is a weight chosen to enhance details by weighting thehigh-frequency component image more heavily. In some implementations,determining 570 the third enhanced image may include checking foroverflow and/or underflow conditions. For example, the third enhancedimage may be determined 570 using the technique 600 of FIG. 6.

It may be advantageous to apply gains associated with the third enhancedimage in the physical domain, before the gamma curve application inorder to avoid splitting the local tone mapping and to take advantage ofa wider dynamic range. In some implementations, determining an outputimage based on the third enhanced image may include: determining gainsfor respective image portions based on the third enhanced image and thegamma curve; and applying the gains for respective image portions tocorresponding image portions of the image. The representation of thethird enhanced image in the physical domain is:

=γ⁻¹( y _(n)+β(y _(n) −y _(n)))where

is the third enhanced image in the physical domain, and γ⁻¹( ) is theinverse transformation of the gamma curve. The detail enhancements ofoperations 530 and 570 may be combined. First determine gains relativeto the enhancement in the physical domain as:

Gphys n = Y ⁡ ( n ) Y ⁡ ( x n )

then the final tone mapping gains, which are the composition of thephysical domain enhancement of operation 530 and the perceptual domainenhancement of operation 570, may be determined as:

Gtotal n = Y ⁡ ( n ) Y ⁡ ( x n ) = ⁢ ⁢ Y ⁡ ( γ - 1 ⁡ ( γ ⁡ ( Gphys n ⁢ ⁢ x _ ) +β ⁡ ( γ ⁡ ( Gphys n ⁢ ⁢ x ) - γ ⁡ ( Gphys n ⁢ ⁢ x _ ) ) ) ) Y ⁡ ( x n )These total gains may replace a global tone mapping gain, and can beapplied on input image in the physical domain (e.g., an RGB formatimage) to determine mapped image.

The technique 500 includes storing, displaying, or transmitting 580 anoutput image based on the second enhanced image. In someimplementations, the output image is the second d enhanced image. Insome implementations, the second enhanced image may by subject toadditional image processing (e.g., perceptual tone mapping with a gammacurve γ( ), lens distortion correction, electronic rolling shuttercorrection, stitching with parallax correction and blending to combineimages from multiple image sensors, electronic image stabilization,and/or output projection) to determine the output image. For example,the output image may be transmitted 580 to an external device (e.g., apersonal computing device) for display or storage. For example, theoutput image may be stored 580 in memory of a processing apparatus(e.g., the processing apparatus 212 or the processing apparatus 262).For example, the output image may be displayed 580 in the user interface220 or in the user interface 264. For example, the output image may betransmitted 580 via the communications interface 218.

The detail enhancement (e.g., as performed at operations 330, 530, and570) incorporated into a local tone mapping might lead to some dynamicrange overflow or underflow. When these dynamic range violations occurnear zero, they are called underflows. When these dynamic rangeviolations occur near the saturation level of the pixel values, they arecalled overflows. For example, underflows can occur when α(x_(n)−x_(n))<−x _(n) for at least one of the color channels. This may result inclipping and associated distortion of the output, where pixel values areconstrained to be positive.

To avoid that, the component image values may be slightly modified whenan underflow condition occurs, meaning that some high-frequencycomponent image will be transferred to the low-frequency component imagewhile preserving the sum of low and high frequency component images.Using this approach, in extreme worst cases, all the energy of a pixelwill be transferred to the low-frequency component image and the tonemapping will locally reduce to the global tone mapping of {acute over(x)}_(n)=λ(x_(n))x_(n), so degradation of the image compared to globaltone mapping only approach may be avoided.

Modifying the results only when the resulting value is below zero couldlead to some discontinuities in the image. To avoid suchdiscontinuities, this modification may be applied and interpolated inrange near zero. (Note: to simplify the equations below, assume that xis a monochromatic image). A threshold μ is specified and energy istransferred between components of the image when |α(x_(n)−x _(n))|<μx_(n). When this occurs, the goal is to modify corresponding imageportions (e.g., pixels) of the low-frequency component image pixel andthe high-frequency component image according to:

= x _(n)+δ_(n)d{grave over (x)} _(n)=(x _(n) −x _(n))=x _(n) −x _(n)−δ_(n) =dx−δ _(n)The following equation may be solved in order to determine δ to preservethe contrast between the enhanced image (after underflow compensation)and the original image:

$\begin{matrix}{\frac{x_{n} - {\overset{\_}{x}}_{n}}{{\overset{\_}{x}}_{n} + \left( {x_{n} - {\overset{\_}{x}}_{n}} \right)} = \frac{\alpha\left( {\left( {x_{n} - {\overset{\_}{x}}_{n}} \right) - \delta_{n}} \right)}{x_{n} + \delta_{n} + {\alpha\left( {\left( {x_{n} - {\overset{\_}{x}}_{n}} \right) - \delta_{n}} \right)}}} & {{Equation}\mspace{14mu} 2}\end{matrix}$Finally, the resulting δ_(n) may be interpolated and applied as follows:

$\begin{matrix}{{{{\hat{\delta}}_{n} = 0},{{{if}\mspace{14mu}{{\alpha\left( {x_{n} - {\overset{\_}{x}}_{n}} \right)}}} \leq {\mu_{1}{\overset{\_}{x}}_{n}}}}{{{\hat{\delta}}_{n} = {\delta_{n}\frac{r - \mu_{1}}{\mu_{2} - \mu_{1}}}},{{{if}\mspace{14mu}\mu_{2}{\overset{\_}{x}}_{n}} \leq {{\alpha\left( {x_{n} - {\overset{\_}{x}}_{n}} \right)}} > {\mu_{1}{\overset{\_}{x}}_{n}}}}{{{\hat{\delta}}_{n} = \delta_{n}},{otherwise}}{{where}\text{:}}{r = \frac{{\alpha\left( {x_{n} - {\overset{\_}{x}}_{n}} \right)}}{{\overset{\_}{x}}_{n}}}} & {{Equation}\mspace{14mu} 3}\end{matrix}$

FIG. 6 is a flowchart of an example technique 600 for high-frequencyenhancement with underflow handling. The technique 600 includes checking610 an underflow condition for an image portion of an enhanced image;where the underflow condition occurs, determining 620 an offset foravoiding the underflow condition; adding 630 the offset to acorresponding image portion of the low-frequency component image; andsubtracting 640 the offset from a corresponding image portion of thehigh-frequency component image; recalculating 650 the image portionbased on the respective adjusted image portions of the low-frequencycomponent image and the high frequency component image; and, when nounderflow condition is detected for the image portion, return 660 theresulting image portion of the high-frequency enhanced image. Forexample, the technique 600 may be implemented by the system 200 of FIG.2A or the system 230 of FIG. 2B. For example, the technique 600 may beimplemented by an image capture device, such the image capture device210 shown in FIG. 2, or an image capture apparatus, such as the imagecapture apparatus 110 shown in FIG. 1. For example, the technique 600may be implemented by a personal computing device, such as the personalcomputing device 260.

The technique 600 includes checking 610 an underflow condition for animage portion (e.g., a pixel or block of pixels) of an enhanced image.In some implementations, a weighted sum used to implement contrast ordetail enhancement is determined (e.g., as described in relation tooperations 430, 530, or 570) and the initial result is compared tothreshold. In some implementations, the threshold is zero (e.g., we onlymodify the enhanced image when the enhanced image would otherwise beclipped due to a dynamic range violation). In some implementations, thethreshold may be set to a small positive value and image portions thatinclude one or more values near or below zero are modified in aninterpolated manner (e.g., as described further in relation to Equations2 and 3) to mitigate discontinuities.

If (at operation 615) an underflow condition is detected, then an offsetδ_(n) for the image portion is determined 620. In some implementations,the offset δ_(n) is determined 620 by iteratively increasing from aninitial value. In some implementations, an interpolated offsetδ_final_(n) is determined 620 (e.g., as described further in relation toEquations 2 and 3).

The technique 600 includes adding 630 an offset to a corresponding imageportion of the low-frequency component image, and subtracting 640 theoffset from a corresponding image portion of the high-frequencycomponent image. For example, energy may be transferred from thehigh-frequency component image to the low-frequency component imageaccording to:

= x _(n)+δ_(n)d{grave over (x)} _(n)=(x _(n) −x _(n)){grave over ( )}=x _(n) −x_(n)−δ_(n) =dx _(n)−δ_(n)where dx is the initial high-frequency component image, d{grave over(x)} is the adjusted high-frequency component image, is x the initiallow-frequency component image, and

is the adjusted low-frequency component image.

The image portion (e.g., a pixel or block of pixels) may then berecalculated 650 based on respective image portions of the adjustedlow-frequency component image

and the adjusted high-frequency component image d{grave over (x)}_(n).For example, for physical domain enhancement, the image portion may berecalculated 650 as {acute over (x)}_(n)=

+αd{grave over (x)}_(n). The new value for the image portion of theenhanced image may then be checked 610 again for an underflow condition.

When (at operation 615) an underflow condition is not detected, theimage portion of the enhanced image is returned.

In some implementations (not shown), the technique 300 may be modifiedto perform a detail enhancement on the image in a perceptual domain.This modified technique may include determining a perceptual domainimage based on the enhanced image and a gamma curve γ( ) that modelshuman perception of contrast; determining a low-frequency componentperceptual domain image and a high-frequency component perceptual domainimage as components of the perceptual domain image; and determining anenhanced perceptual domain image based on a weighted sum of thelow-frequency component perceptual domain image and the high-frequencycomponent perceptual domain image, where the high-frequency componentperceptual domain image is weighted more than the low-frequencycomponent perceptual domain image. When this modified technique is used,the output image that stored, displayed, or transmitted 370 is based onthe enhanced perceptual domain image. In some implementations,determining the low-frequency component perceptual domain image includesapplying a transformation, based on the gamma curve, to a result ofapplying the gains for respective image portions to the low-frequencycomponent image. In some implementations, determining an output imagebased on the enhanced perceptual domain image may include: determininggains for respective image portions based on the enhanced perceptualdomain image and the gamma curve; and applying the gains for respectiveimage portions to corresponding image portions of the image.

It should be noted that the techniques described in relation to FIGS.3-6 and similar techniques may be applied to multiple images fromdifferent image sensors of image capture apparatus (e.g., the apparatus110 shown in FIG. 1) The resulting tone mapped images may subsequentlybe combined using a stitching operation. In some implementations,multiple images from different image sensors of an image captureapparatus may be combined using a stitching operation and the combinedimage, which includes images from each of the image sensors, maysubsequently be tone mapped using the techniques described in relationto FIGS. 3-6 and similar techniques.

While the disclosure has been described in connection with certainembodiments, it is to be understood that the disclosure is not to belimited to the disclosed embodiments but, on the contrary, is intendedto cover various modifications and equivalent arrangements includedwithin the scope of the appended claims, which scope is to be accordedthe broadest interpretation so as to encompass all such modificationsand equivalent structures as is permitted under the law.

What is claimed is:
 1. A system comprising: an image sensor configuredto capture an image; and a processing apparatus configured to: receivethe image from the image sensor; determine a perceptual domain imagebased on the image and a gamma curve that models human perception ofcontrast; determine a low-frequency component perceptual domain imageand a high-frequency component perceptual domain image as components ofthe perceptual domain image; determine an enhanced image based on aweighted sum of the low-frequency component perceptual domain image andthe high-frequency component perceptual domain image, where thehigh-frequency component perceptual domain image is weighted more thanthe low-frequency component perceptual domain image; and determine anoutput image based on the enhanced image, wherein the determination ofthe output image comprises the processing apparatus being configured to:determine gains for respective image portions based on the enhancedimage and the gamma curve, and apply the gains for respective imageportions to corresponding image portions of the image to obtain anoutput image.
 2. The system of claim 1, in which the determination ofthe low-frequency component perceptual domain image comprises theprocessing apparatus being configured to apply a transformation, basedon the gamma curve, to a result of application of a tone mapping to alow-frequency component of the image.
 3. The system of claim 1, in whichthe determination of the low-frequency component perceptual domain imagecomprises the processing apparatus being configured to: apply abilateral filter to the image to obtain a low-frequency component image;and apply a transformation, based on the gamma curve, to a result ofapplying a tone mapping to the low-frequency component of the image. 4.The system of claim 3, in which the processing apparatus is configuredto: determine a reduced resolution image based on the image that is at alower resolution than the image; and in which applying the bilateralfilter comprises processing pixels of the reduced resolution image ascandidates.
 5. The system of claim 3, in which the application of thebilateral filter comprises the processing apparatus being configured to:subsample candidates within a range of distances from a kernel center.6. The system of claim 3, in which the application of the bilateralfilter comprises the processing apparatus being configured to: subsamplecandidates at a first subsampling factor within a first range ofdistances from a kernel center; and subsample candidates at a secondsubsampling factor within a second range of distances from the kernelcenter.
 7. The system of claim 1, in which the image sensor is attachedto the processing apparatus.
 8. A method comprising: receiving an image;determining a perceptual domain image based on the image and a gammacurve that models human perception of contrast; determining alow-frequency component perceptual domain image and a high-frequencycomponent perceptual domain image as components of the perceptual domainimage, in which determining the low-frequency component perceptualdomain image comprises applying a transformation, based on the gammacurve, to a result of applying a tone mapping to a low-frequencycomponent of the image; determining an enhanced image based on aweighted sum of the low-frequency component perceptual domain image andthe high-frequency component perceptual domain image, where thehigh-frequency component perceptual domain image is weighted more thanthe low-frequency component perceptual domain image; and storing,displaying, or transmitting an output image based on the enhanced image.9. The method of claim 8, in which determining the output imagecomprises: determining gains for respective image portions based on theenhanced image and the gamma curve; and applying the gains forrespective image portions to corresponding image portions of the image.10. The method of claim 8, in which determining the low-frequencycomponent perceptual domain image comprises: applying a bilateral filterto the image to obtain the low-frequency component image.
 11. The methodof claim 10, comprising: determine a reduced resolution image based onthe image that is at a lower resolution than the image; and in whichapplying the bilateral filter comprises processing pixels of the reducedresolution image as candidates.
 12. The method of claim 10, in whichapplying the bilateral filter comprises: subsampling candidates within arange of distances from a kernel center.
 13. The method of claim 10, inwhich applying the bilateral filter comprises: subsampling candidates ata first subsampling factor within a first range of distances from akernel center; and subsampling candidates at a second subsampling factorwithin a second range of distances from the kernel center.
 14. A methodcomprising: receiving an image from an image sensor; applying a filterto the image to obtain a low-frequency component image and ahigh-frequency component image; determining a non-linear mapping basedon a histogram analysis of image portions of the of the low-frequencycomponent image applying the non-linear mapping to the low-frequencycomponent image to obtain gains for respective image portions; applyingthe gains for respective image portions to corresponding image portionsof the image to obtain an enhanced image; and storing, displaying, ortransmitting an output image based on the enhanced image.
 15. The methodof claim 14, in which the filter is a bilateral filter, and applying thebilateral filter comprises: subsampling candidates at a firstsubsampling factor within a first range of distances from a kernelcenter; and subsampling candidates at a second subsampling factor withina second range of distances from the kernel center.
 16. The method ofclaim 14, comprising: determining a perceptual domain image based on theenhanced image and a gamma curve that models human perception ofcontrast; determining a low-frequency component perceptual domain imageand a high-frequency component perceptual domain image as components ofthe perceptual domain image; determining an enhanced perceptual domainimage based on a weighted sum of the low-frequency component perceptualdomain image and the high-frequency component perceptual domain image,where the high-frequency component perceptual domain image is weightedmore than the low-frequency component perceptual domain image; andwherein the output image is based on the enhanced perceptual domainimage.
 17. The method of claim 16, in which determining thelow-frequency component perceptual domain image comprises applying atransformation, based on the gamma curve, to a result of applying thegains for respective image portions to the low-frequency componentimage.
 18. The method of claim 16, in which determining the output imagecomprises: determining gains for respective image portions based on theenhanced perceptual domain image and the gamma curve; and applying thegains for respective image portions to corresponding image portions ofthe image.
 19. The method of claim 14, in which the filter is abilateral filter, and applying the bilateral filter comprises:determining a reduced resolution image based on the image that is at alower resolution than the image; and processing pixels of the reducedresolution image as candidates.
 20. The method of claim 14, in whichdetermining the enhanced image comprises: checking an underflowcondition for an image portion of the enhanced image; and where theunderflow condition occurs, adding an offset to a corresponding imageportion of the low-frequency component image, and subtracting the offsetfrom a corresponding image portion of the high-frequency componentimage.